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ABSTRACT 


The advent of Light Detection and Ranging (LiDAR) point cloud collection has 
significantly improved the ability to model the world in precise, fine, three-dimensional 
details. The objective of this research was to demonstrate accurate, foundation methods 
for fusing LiDAR data and photogrammetric imagery and their potential for change 
detection. The scope of the project was to investigate optical image to LiDAR 
registration methods, focusing on several dissimilar image types including Optical Bar 
Camera (OBC), high resolution aerial frame, and WorldView 1 satellite with varying 
LiDAR point densities. An innovative optical image to LiDAR data registration process 
was established. This approach was demonstrated for one image type using the rational 
polynomial coefficients (RPC) representation of the panoramic math model improving 
accuracy from 1.9 m to 0.5 m root mean square (RMS) error. Comparison of stereo 
imagery point cloud data to the LiDAR point cloud using a 90% confidence interval 
highlighted changes that included small scale (< 50cm), sensor dependent change and 
large scale, new home construction change. This research also proposed a fused LiDAR 
and stereo image base layer as the foundation for further LiDAR/image fusion. 
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I. 


INTRODUCTION 


Light Detection and Ranging (LiDAR) uses the measurement of timed light 
pulses to determine elevation differences. It produces “point clouds” of 3D data of 
varying density depending on the light source and interaction with the Earth’s surface. 
LiDAR point cloud data have an outstanding ability to represent and portray topography, 
building structures, and foliage with precise, fine, three dimensional details. (Shan & 
Toth, 2009) The precision required for Light Detection and Ranging (LiDAR) point 
cloud data to successfully model the earth’s surface and features is an open question. The 
goals of LiDAR measurements are similar to those of Photogrammetry. In 
Photogrammetry, the objective is to extract precision three-dimensional representations 
of the earth’s surface using analysis of parallax in stereo images. Even when viewed as 
photographs, photogrammetric images can appear precise because of the many fine and 
structured details that are visible and interpreted. LiDAR uses direct measurement of 
distances with respect to known ground control to determine point clouds that can be 
further analyzed to generate Digital Surface Models (DSM) or Digital Elevation Models 
(DEM).The nature of these two data sets makes them ideal for improved 3D 
representation of surfaces. To successfully accomplish this, however, the precision and 
inherent errors of each dataset need to be well understood. 

It is common for LiDAR point cloud data and Photogrammetric quality imagery 
(mono and stereo) to be collected at different times over an area of interest. This 
difference in collection times can span several months to years. This fact makes co¬ 
registration and fusion challenging. 

This research demonstrates the processes and results of fusing LiDAR point cloud 
data with photogrammetric imagery and derived stereo photogrammetric image point 
cloud data. The process highlights innovative techniques of registering Optical imagery 
to LiDAR data. In addition, the process demonstrates the improved registration accuracy 
when Rational Polynomial Coefficients (RPC) support data are used in place of the 
rigorous panoramic model for the Optical Bar Camera (OBC). Once the stereo optical 

imagery are coincident with the LiDAR data, fusion of optical imagery with LiDAR point 
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cloud data and fusion of stereo image derived point cloud data and LiDAR point cloud 
data can be accomplished. 

We usually think about data fusion as a resource to answer a specific question or 
as a tool to tackle a specific problem. Data fusion can also be a means to create a new 
data set that is more readily exploitable. Several examples of using data fusion to create 
a more useable dataset exist. For example, the fusion of multispeetral and panchromatic 
imagery is used to create a “pan-sharpened” or “resolution merge” image (Pohl & Van 
Genderen, 1998). Usually the multispeetral image is lower resolution than the 
panchromatic image and the goal is to create a true color image from the multispeetral 
image and then increase the resolution of this true color image by using the panchromatic 
image in a sub-pixel sampling process. This process has become so common that we 
often do not think about it as a data fusion process. Pan-sharpening imagery is an 
elegant example of how fusion of data sets can make a new data set that is more useable 
and ready for exploitation. 

Taking the above example a step further, elevation data can used to create an 
orthophoto of the pan-sharpened image. This three source data fusion process of ortho- 
rectifying pan-sharpened color imagery creates a base layer of imagery that can be map 
quality and useable for precise feature extraction. We exploit the results of this three 
source fusion process every day when we open Google Earth™ or Yahoo Maps™. 

The purpose of these examples was to show how geospatial data can be fused to 
create more useable and exploitable data sets. This fusion process can extend into other 
remote sensing phenomenologies sueh as LiDAR and Hyperspeetral Imagery, Radar and 
Optical Imagery, and LiDAR and Optical imagery. The focus of this research was to 
study the results of fusing LiDAR point cloud data with Optical imagery and Stereo 
Optical imagery derived point cloud data. 
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II. BACKGROUND 


A. HISTORY OF THE PROBLEM 

LiDAR point cloud data can contain a lot of information in 3D space. If dense 
enough, this can be very useful at the individual feature level and can provide realistic 
views that can be rotated and viewed from many angles (Figure 1). 


MSL Height 
^ - 36 m 


I - 25 m 
—J - 14 m 


Scale: 


15 m 


30 m 


Location: 36.596 N 121.874 W 




Figure 1. LiDAR point cloud data view showing individual 3D feature level 
information. Individual points are color coded by 
Mean Sea Level (MSL) height. 


The application of laser ranging and point cloud collection has exploded in the 
last decade. Close range modeling, terrestrial navigation, and airborne topography are 
varied examples of problems that continue to drive a need for precision LiDAR point 
cloud data. The fundamentals of using a laser to collect ranging or distance information 
in large volume apply to many 3D knowledge problems. 
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The history of LiDAR and photogrammetry fusion shows that many individuals 
with diverse interests are studying this problem. Experts from the Computer Vision 
community, Photogrammetry community, and LiDAR community all have an interest in 
solving the problems of LiDAR and imagery fusion (Kaminsky et ah, 2009; Schenk et ah, 
2001; Rodarmel et ah, 2006). After comprehensive study, these diverse disciplines show 
significant overlap and the problems to be solved are similar. No matter which one of 
these disciplines chosen, rapid collection of precision point cloud data and extraction of 
the 3D information content of that data are the goals. 

Registering LiDAR collected point cloud data to optical imagery is not as easy as 
it sounds because point cloud data contains discrete points that are not easy to precisely 
locate on an optical image. Lor instance, on an optical image with the proper resolution, 
the comer of a sidewalk or building or roof top can be precisely identified and measured. 
With LiDAR point cloud data, the exact comer of a building usually does not contain a 
collected point. This presents a dilemma for the registration of these two data sets. 

Two prevailing methods have been developed to overcome this registration 
dilemma (Mishra & Zhang, 2012). The first method uses derived edges from both data 
sets and attempts to match them up. The second method uses the LiDAR derived 
intensity image and DEM data to control the optical image. 

Computer Vision solutions to this fusion problem have grown in popularity. 
Many reports elaborate on how computer vision multi-image matching has stepped up in 
place of photogrammetry to build point clouds (Leberl et ah, 2010). One Computer 
Vision approach to the problem is to build the point cloud from numerous overlapping 
images in an arbitrary image model space and then align this point cloud to imagery 
through classic point, edge, and homogeneous area matching (Kaminsky et ah, 2009). 

While Computer Vision solutions are new and attractive, they rely heavily on 
image content such as building structures and roads that have strong edges and features to 
perform image and point cloud matching . The idea of using linear features for matching 
is also prevalent in the photogrammetry community as it has been successfully 
implemented in image to image matching routines (Habib, Ghanma, & Kim, 2005). A 
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quick visualization of an optical image and LiDAR point cloud shows the strong edges of 
both data sets that would make ideal matching features (Figure 2). 



MSL Height 
^ - 36 m 


- 25 m 


Figure 2. Side-by-side view of a high resolution color frame image (left) 

and top view of a LiDAR point cloud (right) highlighting linear edge 
features ideal for automatic registration. 


Areas where topography is being derived, unfortunately, do not always contain 
urban structure, and tend to be very homogeneous in image content. As a result, 
topographic and mapping applications tend to lean toward using the intensity image and 
LiDAR derived Digital Elevation Model (DEM) as a reference base. This solution for 
registration is not popular, however, because it is not well suited for automation (Mishra 
& Zhang, 2012). 

One registration approach encountered is to create two images from the LiDAR 
data (Habib, Ghanma, & Kim, 2005). One image is the traditional intensity image and 
the second image is the DEM represented as a “range” image. This does not seem very 
accurate though because the height value is fixed for any given sub-pixel location. A 
better approach would be to interpolate a height value between the DEM points. 

Many traditional photogrammetry researchers have recognized the mutual 

benefits that a photogrammetric derived point cloud and a EiDAR point cloud can 
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provide (White et ah, 2013; Habib et al, 2008). The problem of fusing LiDAR data and 
photogrammetric imagery becomes much more complex when registering large-scale 
imagery and attempting to create true orthophotos (Habib et al, 2008). True orthophotos 
are different than a terrain corrected orthophoto. True orthophotos use detailed 3D 
feature information to stand buildings up. In the typical terrain corrected orthophoto, 
buildings and tall features will still retain the perspective view from the photogrammetric 
image often showing lean away from the center of the image. The Digital Surface Model 
(DSM) can be more completely described with the fusion of the LiDAR point cloud and 
stereo point cloud, resulting in better representation of the 3D environment. This 
improved representation of the 3D environment can produce more precise true 
orthophotos. 

Outside of the urban environment, forest management has started to engage the 
utility of LiDAR point cloud data and how it can be used with the more traditional aerial 
photogrammetric practices today (White et al., 2013). The partial tree canopy penetration 
capabilities of LiDAR help to establish tree height and tree maturity. Also, the 
underlying Digital Terrain Model (DTM) can be created from the LiDAR data to 
understand water drainage and road access. These data can then be fused with 
photogrammetric and multispectral image derived tree species and tree health 
information, resulting in a complete model of the forest inventory. 

LiDAR collected point clouds and stereo-image-derived point clouds can also be 
used for change detection (Schenk & Csatho, 2003), which is one of the focuses of this 
research. The derived data from each sensor can be used to investigate catastrophic 
change that may have occurred as a result of a natural or man-made disaster. Rapid 
assessment of destructive events allows local emergency response and local government 
personnel to make better actionable decisions. 

Another concept in LiDAR and optical image fusion is the ability to use accurate 

stereo imagery to assess the quality of LiDAR point clouds. In layman’s terms, this is 

sometimes known as “draping” the point cloud over the stereo images. In 

photogrammetric practice, a technique called “back-projection” is used, which takes the 

X,Y,Z coordinate of each point in the point cloud and computes the corresponding image 
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coordinates for each point (Schenk, Seo, & Csatho, 2001). The points can then be 
viewed in stereo, comparing the point locations to actual surface features. If the stereo 
images have a reported accuracy, then this technique can be used to establish or verify 
LiDAR point cloud accuracy. This technique is frequently used for Digital Elevation 
Model (DEM) evaluation. 

The EiDAR point cloud and single optical image fusion process is the most 
intriguing near term topic in terms of what it has to offer towards improved surface 
analysis. Standard practice for deriving precision 3D coordinates is to use accurate stereo 
imagery and a two-ray image measurement intersection routine. Performing a stereo 
measurement can be very challenging for an analyst and requires proper viewing 
equipment. A fused EiDAR-optical image data set allows precision computation of 3D 
coordinates without relying on this stereo image measurement. The process of deriving a 
3D coordinate on a single image is called “single ray intersection.” The single ray 
intersection process needs precise elevation data coincident with the single image in order 
to correctly intersect the image ray on the ground or object. The EiDAR data registered 
to the image can provide these precise elevation data for objects or ground features in the 
image allowing for precise 3D coordinate derivation through the single ray intersection 
process (Rodarmel et ah, 2006). 

Being able to compute precise 3D coordinates from a single image has advantages 
for improved exploitation. Single image exploitation can be done on a handheld device 
like a cell phone or tablet. The process of exploitation could be as simple as placing a 
cursor on a feature in the image. This research explores the EiDAR and optical image 
fusion process starting with the problem of image and EiDAR registration. 
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III. OBJECTIVES 


The main objectives of this research was to demonstrate innovative techniques of 
the LiDAR point cloud and photogrammetric image registration process and to compare 
LiDAR point cloud and stereo photogrammetric derived point clouds, their 
characteristics, and precision. This thesis also summarizes the important points of 
creating a fused image/LiDAR data set for improved exploitation. 
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IV. DATA PROCESSING METHODS, RESULTS, AND ANALYSIS 


A. STUDY AREA AND DATA 

1. Haiti LiDAR and Optical Bar Camera Imagery 

The first study area was chosen near Port-au-Prince, Haiti because both LiDAR 
data and Optical Bar Camera (OBC) imagery existed as a result of support for earthquake 
and humanitarian relief efforts. Both data sets were collected in late January of 2010 
following the 7.0 magnitude earthquake that occurred near Port-au-Prince, Haiti on 
January 12, 2010. A small subset of LiDAR data was selected from a larger available 
LiDAR database that coincided with the area of the OBC image coverage. The following 
map graphics show the exact location of the LiDAR subset (Figure 3). 



Scale: 

czzz 


10 km 

3 - 


20 km 


Scale: 2.5 km 5 km 

I ■ —I 

E Location: 18.534 N 72.469 W 


Figure 3. This overview map (left) and zoomed map (right) show the area in Haiti 
near Port-au-Prince where the LiDAR data were extracted from the 
OpenTopography Website. The red box in the right zoom map display 
highlights the location of the downloaded LiDAR data (3.4 points/m^). 


The LiDAR data collection was performed by the Center for Imaging Science at 
Rochester Institute of Technology (RIT) and Kucera International under sub-contract to 
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ImageCat, Inc. (Open_Topography, 2008). The downloaded data eonsisted of 
28.5 million points with a density of 3.4 points/m . The project was funded by the Global 
Facility for Disaster Reduction and Recovery (GFDRR) and the data are hosted at the 
World Bank (www.opentopography.org). The following figure (Figure 4) shows a 3D 
perspective view color coded by Above Ground Level (AGL = DSM-DEM) height of the 
downloaded Haiti LiDAR data and displayed in Quick Terrain Modeler (QTM) 
(Appliedjmagery, 2004) . 



Figure 4. Oblique view of downloaded Haiti LiDAR data color coded by 
Above Ground Level (AGL) Height (in meters). This data set 
eontains approximately 28.5 million points. 


The Optical Bar Camera (OBC) imagery is collected by the United States Air 
Force (UTC_Aerospace_Systems, 2013). The OBC itself is a large format calibrated 
panoramic camera capable of photogrammetric quality image eollection. The OBC 
imaging mission for Haiti was performed on January 30, 2010 flying at an altitude of 
42,000 feet producing an approximate Ground Sample Distanee (GSD) of 0.15 meters. 
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The OBC imagery is film (hardcopy) based and must be digitized. The large 
format image is traditionally digitized using 14 micrometer pixel sizes in 15 degree, 8-bit 
TIP or NITF format segments. One segment was chosen that covered the LiDAR test 
area (Figure 3 and Figure 5). This OBC segment is centered on the -i-15 degree mark of 
the full (-72 to -1-72 degree) panoramic image and was further clipped to eliminate the 
water area as no control points could be selected in that area. 



Scale: .75 km 1.5 km 


Location: 18.534 N 72.469 W 



Figure 5. Segmented piece of OBC imagery in Haiti near the LiDAR 
coverage area. This image was further clipped to eliminate 

the water. 


2. NFS Campus LiDAR and High Resolution Aerial Imagery 

High resolution LiDAR and optical data were recently collected over a portion of 
the Naval Postgraduate School campus, Monterey, California, and are included as part of 

this study. The LiDAR point cloud was collected in October 2012 by Watershed 
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Sciences, Inc. (WSI). Mounted in a Bell 206 Long Ranger Helicopter, the LiDAR data 
were collected at a very dense average of 30 points/m^ using an Optech Orion C200 laser 
system (Watershed Sciences, 2012). The following graphic shows the location of the 
complete LiDAR collect over the Monterey Peninsula. A subset of this data over the 
NPS campus was selected as the focus of this thesis (Figure 6, Figure 7). 


Location of NPS Campus Subset 

_ J. _ 



Location: 36.596 N 121.874 W 


Figure 6. Image from the Watershed Sciences Report (Watershed Sciences, 2012) 
showing the complete Monterey peninsula WSI LiDAR collection 

in October, 2012 


14 








MSL Height 
36 m 

- 25 m 

I— I - 14 m 



Location: 36.596 N 121.874 W 


I 


Figure 7. Selected LiDAR subset of the NFS campus. 3D Perspective view 
displayed in QT Modeler software and colored by height 
(internally known as tile 774) 


The optical imagery was collected in stereo using a multispectral UltraCam Eagle, 
260 megapixel camera manufactured by Vexel Corporation, a Microsoft subsidiary. The 
color imagery scans were collected at 450m flying height producing high resolution 15cm 
pixels (Watershed Sciences, 2012). The following diagram (Figure 8) shows the 
footprints of the single images, the amount of overlap between successive images, and 
the amount of sidelap between the strips. The NFS campus area is near the center, 
marked by the red outline. 
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Figure 8. UltraCam Eagle stereo image footprint coverage diagram showing overlap 

between images and sidelap between strips. The NFS campus area is marked by 

the red outline. 


The imagery has very good photogrammetric quality capable of supporting 
precision DEM extraction. The following split screen view (Eigure 9) of the UltraCam 
Eagle color images shows a section of the NFS campus. 
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Figure 9. Stereo images of the NFS campus shown in split screen view detailing the 
quality of the stereo imagery capable of precise, high resolution DEM or point 
cloud extraction. The red lines highlight part of the DEM extraction area. 

3. San Diego LiDAR and WorldView 1 Imagery 

The third study area was chosen near Pacific Beach, California, which is just 
north of San Diego, California. This study area was chosen because WorldView 1 
(WVl) stereo imagery was available along with EiDAR data (Open_Topography, 2008). 
A small subset of EiDAR data was selected and downloaded using the opentopography 
web site EiDAR retrieval tool. The EiDAR was collected in January 2005 by Merrick, 
Inc. for the City of San Diego at 1.4 points/m . This small subset of EiDAR data 
coincided with the area of the WVl image coverage. The following map graphics 
(Eigure 10) show the exact location of the EiDAR subset. 
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Figure 10. This overview and zoomed map display show the area in 

Pacific Beach near San Diego, California where the LIDAR data were 
extracted from the OpenTopography Website (Open_Topography, 2008). 
The red box in the right zoom map display highlights the location of the 
downloaded LiDAR data (1.4 points/m ). 
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Figure 11. Colorized rendition of a zoomed in section located in the Southeast 
corner of the downloaded Pacific Beach LiDAR data set. 

The data was collected in 2005 at a 1.4 points/m^ density. 


The stereo optical imagery used in this third case study is from Digital Globe’s 
WVl satellite. WVl is strictly a panchromatic (BAV) image collector with large 
footprints and 0.6m pixels. The images were collected as a stereo triplet—3 images 
consecutively. Each stereo pair image combination (1-2, 1-3, 2-3) from the WVl triplet 
was used to extract a DEM using photogrammetric methods, and from these three 
independent DEMs, the one with the least amount of automatic correlation errors was 
used. The following graphic (Eigure 12) shows the coverage of the WVl stereo triplet as 
well as the outline of the EiDAR coverage. 
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Figure 12. This image shows the large red outline of San DiegoWVl stereo 
triplet coverage and a smaller interior red box that outlines the 
downloaded LiDAR coverage for the Pacific Beach area. 


The following images (Figure 13) show a split screen view of the WVl stereo 
imagery detailing the quality and perspective view of the individual images. 


20 






Figure 13. WV-1 stereo image area of Pacific Beach shown in split screen view 

detailing the quality of the stereo imagery capable of DEM or point cloud 
extraction. (GSD = 0.6 meters) 

B. REGISTRATION OF OPTICAL IMAGERY TO LIDAR 

This research focused on using the “Intensity Image” approach for controlling 
optical imagery to LiDAR (Mishra & Zhang, 2012) as opposed to the feature or edge 
matching approach that has been thoroughly studied by several researchers (Du, Xu, & 
Cao, 2009; Mishra & Zhang, 2012). In the majority of cases, the identifiable feature 
content of an intensity image will be sufficient enough for common points to be found 
between the optical image and the intensity image. It is also important to select these 
common points in relatively flat regions because the height value for the point selected on 
the intensity image will have to be interpolated from the LiDAR created TIN or DEM. If 
the point selection is done properly, very accurate points can be interpolated from the 
LiDAR data even though the selected point may not represent a real collected LiDAR 
point. Once collected, these precision interpolated points can be used to control the 
optical imagery in a triangulation process. 
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The following process flow summarizes a standard method for controlling optical 
imagery to LiDAR data (Table 1). This generic process was derived from similar image- 
to-image registration processes using the “Socet Set” software (BAE_Systems, 1990). 


Generic LiDAR Control Process Flow 

1. Import LiDAR data (.las file) 

A. Creating Intensity Image 

B. Creating DEM (no thinning) 

2. Import Imagery to be controlled 

3. Build Control Point File using Control Point Editor 

A. Locate Control Points on Intensity Image 

B. Interpolate Height using LiDAR DEM (Terrain Tracking) 

C. Capture a description of control point location (sketch) 

4. Create Triangulation File 

A. Include Image and Control Point File 

5. Locate and Measure each Control Point 
A. Use sketch for correct identification 

6. Use Triangulation Solve to adjust Imagery 


Table 1. Generic LiDAR Control Process Flow Defined Using Socet Set 


As part of this research an innovative, improved process was developed to control 
imagery to LiDAR (Table 2). 
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Improved LiDAR Control Process Flow 

1. Import LiDAR data (.las file) 

A. Creating Intensity Image 

B. Creating DEM (no thinning) 

2. Import Imagery to be controlled 

3. Build Triangulation File 

A. Include Imagery and Intensity Image 

4. Measure Common Points 

A. On LiDAR Intensity Image and Optical Imagery 

B. Point Pattern Sufficient for Image Type 

5. Create Control Points from LiDAR Data Using Triangulation Solve 

A. Solve for ground point locations on intensity image 

B. Update Ground Control File 

C. Interpolate Height using LiDAR DEM 

D. Update ground control file \A/ith new height 

6. Use Triangulation Solve to adjust Imagery 


Table 2. Improved LiDAR Control Process Flow Derived Using Socet Set 


The improved LiDAR control process contains two enhancements. First, 
including all imagery in the Triangulation allows common points to be measured on all 
imagery in the same viewing window and allows the review of those measured points at 
any time before or after the triangulation solution. The capability to review points is very 
useful in the case of control point miss-identification. Second, by activating/deactivating 
images, changing weights on image parameters, and changing weights on ground control 
points, the triangulation solve can be used to derive ground control points in one step and 
then used to adjust the optical imagery in a second step. Using the triangulation solve to 
generate ground control also gives a triangulation report that details residuals and 
parameter adjustments that can be used for quality assurance. 
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1. Haiti LiDAR and Optical Bar Camera Imagery 

Once the basics of registering optical imagery to LiDAR were outlined as above, 
the first challenging case study was to investigate the registration of Optical Bar Camera 
(OBC) imagery to LiDAR for the Haiti site. The registration of OBC imagery using 
control points is challenging. The challenging part is that the imagery were collected 
with an airborne panoramic camera, which introduces the complicated dynamics of the 
panoramic camera and, in the case of OBC, disconnected sensor positioning and 
orientation information. Also, OBC imagery starts out as a hardcopy film that must be 
digitized. Importing this scanned imagery into photogrammetric exploitation software 
using the correct panoramic math model and proper metadata can be tedious. 

The required sensor position and orientation information is separate from the 
imagery and must be correctly identified and captured. Once this important data is 
retrieved, the values must be hand entered in required fields upon image import. The 
example below is from the Socet Set software (Figure 14). 



Figure 14. Socet Set panoramic camera position and orientation input page. 

All data must be input by hand. 
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Because the scanned imagery is a segment of a full panoramic image that may 
occur at any degree sweep, the input roll angle of the sensor has to be changed to give the 
impression that the camera is pointing in that direction (Figure 15). 



Figure 15. Diagram of a 15 degree scanned segment of OBC film centered 

at the negative 15 degree scan mark. 

If the metadata are correctly input, the imagery will import properly and align to 
within 100 meters of absolute ground truth. In any case, this raw OBC image positioning 
otherwise known as “apriori positioning” requires controlling or registration for LiDAR 
fusion and exploitation. At this juncture, the LiDAR data are also imported, creating an 
intensity image and a regularly spaced gridded DEM interpolated from the LiDAR point 
cloud. All of the ingredients are now in place to continue the optical image to LiDAR 
registration process as outlined in Table 2. 

The next step in the process is to measure common points between the OBC 
imagery and the LiDAR intensity image using an interactive point measurement function. 
Selecting and measuring quality common points are a skill that requires practice. The 
goal in optical image to LiDAR intensity image point transfer is to find small features 
that are clearly identifiable in both images. Also, because we need to interpolate a height 
value from the LiDAR DEM, the selected point should be in a relatively flat area. Figure 
16 shows an example of a common selected point. 
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Figure 16. This side-by-side image display of a subset of the Haiti data shows an 
example of a common point selected (in center) between the optical image 
( Left : OBC) and the LiDAR intensity image (Right : Haiti LiDAR Intensity 
Image). Points should be selected that are clearly identifiable and do not have 
variation in surrounding height values. 

Once all the common registration points were measured in a pattern that covers 
the image and dense enough for sufficient redundancy to solve the adjustment parameters 
(see Figure 17 for example of a sufficient pattern), the control point X and Y (Latitude 
and Longitude if working in geographical space) were derived from the LiDAR intensity 
image. This was accomplished by performing a single image triangulation adjustment 
with the image parameters held fixed and the Z ground space coordinate held fixed. The 
Z height value is an arbitrary value at this point and it does not matter what the value is 
because the LiDAR intensity image is an orthoimage. 

After the initial X and Y values for the control points were determined from this 
first triangulation solution, the Z height value was updated using the control point editor. 
The correct Z height value was determined by turning on a terrain tracking tool, selecting 
the LiDAR DEM as the base terrain layer, systematically visiting each control point X 
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and Y location, and recording the Z height value. This step completed the creation of full 
X,Y,Z control points to be used to register the OBC image. 

The final triangulation solution that adjusted the OBC imagery using the rigorous 
panoramic math model parameters produced alignment to the LiDAR point cloud data 
with marginally acceptable accuracy (Figure 17). The Root Mean Square (RMS) of the 
ground residuals was 1.9 meters with the longest residual at 3.0 meters. This corresponds 
to several pixels of misalignment for the image and LiDAR data. 

Previous to this research, investigation had been done to develop more robust and 
user friendly support data for OBC imagery that would have more utility and be able to 
be used across many software packages, not just Socet Set. Success had been achieved in 
computing RPC support data for each segment, which made exploitation much easier. 
An added benefit surfaced as a result of the comparisons between the rigorous model 
support data and the RPC support data. The added benefit of the RPC support data was 
that when the RPC adjustable parameters were used to adjust the imagery, a better fit to 
control was achieved. 

Because of this previous, unpublished knowledge, RPC support data was 
generated for the Haiti segment using the Socet Set “fast sensor model” and “image 
chipping tool.” The same control point selection approach was then used to register OBC 
to the LiDAR with very good results (Figure 17). The RMS of the ground residuals 
improved to 0.5 meters with the longest ground residual at 1.1 meters. 
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Figure 17. These base images for Haiti show a scanned segment of an OBC image 

and the control point locations (red circle with red diamond) with red vectors. 
The Left image shows ground space residuals as a result of the rigorous 
panoramic model adjustment. (Left: RMS 1.9 meters) 

The Right image shows the improved ground space residuals as a result of using 
the RPC adjustment (Right: RMS 0.5 meters). Scale: Longest vector is 1.1 

meters on the right image. 


Following this RPC parameter adjustment aligning the OBC imagery to the 
LiDAR point cloud, an orthorectification of the OBC imagery was performed using the 
adjusted RPC parameters and the LiDAR DEM. QT Modeler software was then used to 
merge the LiDAR point cloud and the OBC orthophoto in a fusion process. The process 
in QT Modeler simply imports the LiDAR data and then adds the orthophoto as texture. 
Internal to QT Modeler, the LiDAR data is stored as individual XYZ points with each 
point receiving the added attribute of a Red-Green-Blue (RGB) value derived from the 
OBC orthophoto at that pixel location. The following figure (Figure 18) shows the 
results of the LiDAR point cloud and OBC image fusion using QT Modeler. 
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Figure 18. OBC image and LiDAR point cloud fusion showing the complete 

registered image. LiDAR points are color coded by Above Ground Level (AGL) 

height where AGL = DSM - DEM. 


The following figure (Figure 19) shows a detailed perspective view using QT 
Modeler to display the LiDAR point cloud with the OBC image used as texture for the 
points. 
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Figure 19. Haiti LiDAR data with the registered OBC used as 

texture for the points 


Figure 19 demonstrates that the building tops appear to have good alignment and 
that the road and tower on the hill also have good alignment with the OBC image texture. 
Individual points from the point cloud can also be seen on the figure. 


2. NFS Campus LiDAR and High Resolution Aerial Imagery 

The NFS Campus LiDAR and High Resolution Aerial Imagery were collected in 
conjunction with precision GPS and INS/IMU data to meet accuracy requirements. 
Because of this accuracy requirement, no registration was performed to adjust the 
imagery to the LiDAR point cloud data. 


3. San Diego LiDAR and WorldView 1 Imagery 

The WVl stereo image registration to LiDAR used the same LiDAR “Intensity 
Image” and LiDAR DEM interpolation technique to create control points that were used 
in a WVl only triangulation process. Refer to Table 1 above for the high level process 
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flow. The following diagram shows the loeation of the control points in the area of the 
LiDAR data set. 



Scale: 2 km 


4 km 

□ 


Location: 32.810 N 117.257 W 


Figure 20. WVl background image (image 214) showing the location of the 

LiDAR boundary in red together with the location of the control points 
(12 points identified as red triangles within red circles) 


Control points on the LiDAR intensity image were selected in relatively flat, 
elevation homogeneous regions. Many of the points were small, flat features in the 
center areas on roof tops as this test area was primarily urban. The following example 
shows the selection of a control point on a street marking. Generally, painted street 
markings do not make good image to image registration control points because they can 
change or be moved slightly over a short time span. This one appeared to be relatively 
unchanged. 
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Figure 21. This side-by-side image display shows an example of a eommon point 
seleeted (center red crosshair) between the optical image (Left: WVl) 
and the LiDAR intensity image (Right: Pacific Beach LiDAR Intensity Image). 

Points should be selected that are clearly identifiable and have very little variation 

in surrounding height values 

Once all the control points are selected, the horizontal ground coordinates for the 
points on the intensity image must be computed. For the purposes of this research, this 
was accomplished using the Socet Set solve routine and adjusting only the X and Y value 
of the point. It could also be computed in other software using similar approaches. The 
image position parameters are “locked” and the Z coordinate (meaningless in this initial 
adjustment) is also “locked.” Running the solve with these settings and only the intensity 
image active will give precise horizontal coordinates for the points that now become 
control points. 

The vertical control value is arrived at through interpolation of the LiDAR DEM, 
as the point that has been created does not coincide with a real point in the LiDAR point 
cloud. In Socet Set, this can be accomplished by turning on terrain tracking with the 
LiDAR DEM selected and systematically visiting each point. Similar approaches can be 
used in other software. This will allow interpolation of a Z value for the visited control 
X,Y location. These interpolated Z values now become the precise vertical control 
location. 
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The final controlling adjustment on the WVl stereo images is performed using 
these control points and a normal, WVl only adjustment. Results of this adjustment were 
very good as shown in Figure 22. Ground control points fit to an RMS of 0.6 meters. 



Figure 22. WVl background image showing ground space residual analysis results 

of the triangulation adjustment that controlled the 
WV-1 stereo triplet to the LiDAR data. The RMS was 0.6 meters 

C. STEREO IMAGE POINT CLOUD EXTRACTION 

The next critical step in this process was to extract a digital surface model (DSM) 
from the stereo images. Socet Set has an automated dense terrain extraction tool called 
the Next Generation Automated Terrain Extraction (NGATE) tool. NGATE has many 
features that allow dense DSMs to be automatically measured that resemble a EiDAR 
point cloud. One feature is the interpolated estimate or starting point of adjacent points. 
The software will use surrounding point correlation information to better predict the 
height to try an auto-correlated measurement. A second feature is the ability of the 
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software to take advantage of multiple (more than 2) perspective views of a single point. 
This can work well around buildings and occluded objects. It should be possible to use 
similar approaches to obtain similar results using other DSM-generation software. 

1. Haiti Site (OBC) 

No stereo data were available for the Haiti site, thus a stereo point cloud was not 
attempted. 

2. NFS Campus Stereo Imagery 

The NFS campus stereo imagery is very good quality photogrammetric imagery 
as reported by the WSI report (Watershed Sciences, 2012) and as shown in Figure 8 and 
Figure 9 above. The stereo pair that covered the “774” subset of the NFS campus area 
was image id 105 and 106. These images have an average of 0.12 meters ground sample 
distance (GSD). NGATE was set up to produce a grid spacing equal to that of the 
LiDAR collection at 0.15 meters. The following figure (Figure 23) shows an example of 
the DEM grid spacing. Even though this is a very small area, the DEM grid contains 
4,020,025 posts. Every fifth post has been displayed in order to see the grid better. 
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Figure 23. Portion of the DEM grid (every 5'*’ post displayed) near Herrmann Hall 

on the NPS eampus showing the density of the grid 
(0.15 meter point spacing) 

Fortunately, the NGATE software in Socet Set will attempt autocorrelation on all 
points as it would be virtually impossible to edit each point manually. The results of the 
collected NGATE autocorrelation DEM is shown in Figure 24. It should be possible to 
generate similar results using other autocorrelation software. 
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Figure 24. High density DEM (0.15 meter point spaeing) over NFS campus 

auto collected from stereo pair 105/106 using Socet Set NGATE software 
(4,020,025 total points). These data were be used for EiDAR/Optical fusion. 
MSE = Mean Sea Eevel heights (MSE = Ellipsoid Height - Geoid Separation) 


The next step in the process was to take the high density DEM and convert it to a 
point cloud. This was accomplished by exporting the DEM from Socet Set as an ASCII 
grid file and importing the ASCII grid file into QT Modeler as a point cloud. The 
following figure (Eigure 25) shows the high density collected DEM converted to a point 
cloud. Other software should be able to achieve similar results. 
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Figure 25. High density DEM (0.15 m. post spacing) imported into QT Modeler 

as a point cloud to be used for LiDAR fusion. 


At first, it appeared as though there were too many errors around the trees and 
near the building edges of the NGATE autocorrelation. The auto-correlated points in the 
tree and building edge areas seemed to be interpolated and of poor quality. Including 
3 images (104/105/106) in the stereo extraction was attempted to aid in precise 
autocorrelation. The results did not, however, improve the stereo point cloud as well as 
expected (Eigure 26). Many autocorrelation errors still existed. The autocorrelation 
errors were also affected by sun shadows even though these images were all shot within 
tens of seconds of each other, and the shadows did not change. 
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Location: 36.596 N 121.874 W 


Figure 26. Side by side view showing the difference between a 3-photo stereo 
extraction (Left Image) and a 2-photo stereo extraction (Right Image). 
Autocorrelation errors exist (10 meters) around trees and near building 
edges in both images with no discernible advantage for either method. 

The two photo stereo point cloud was considered acceptable for use in the LiDAR 
fusion process because all other stereo image autocorrelation results were similar. A 
realization did surface from this experiment. It was concluded that any level of stereo 
DEM extraction would contain errors. The consequence is that the data have to be 
manually edited by measuring each point individually or using tools to edit points as a 
group. This can be extremely time consuming and expensive. The alternative is to 
accept the errors, and use the data in the unedited state—pure autocorrelation data— 
which is the path that I chose for this research. 

3. San Diego WV-1 Stereo Imagery 

The WV-1 stereo imagery over the San Diego area (Figure 12 and Figure 13 
above) was collected as what is called a stereo triplet. This stereo triplet is three images 
collected consecutively in the same orbital pass over an area. WV-1 imagery when 
collected has absolute position capability to 6 meters 90% of the time and each image is 
independent. The fact that the images are independent implies that the stereo pairs will 
have parallax that must be removed. This removal of parallax and the absolute 
positioning improvement was accomplished in one step when all three WV-1 images 
were registered to the FiDAR data (Figure 22 above). The following figure (Figure 27) 
shows the perspective views of the stereo triplet. 
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Figure 27. Side by side display of the WV-1 triplet zoomed in on a prominent 

building feature. The images were collected during the same orbital pass 
on 08 Feb 2009. From left to right the image id’s are 139, 214, and 228 

Images 214 and 228 were chosen for stereo image DEM extraction because the 
perspective views were similar resulting in less occluded features. Figure 27 
demonstrates that image 139 has a perspective view that is completely different than 
image 214 and 228. Using image 139 with any other image would prove to be very 
difficult for the autocorrelation routine. 

The grid spacing for the WV-1 stereo image extracted DEM was set similar to the 
EiDAR data point spacing at 1.4 points per meter. This translated into approximately 0.8 
meter equivalent grid spacing for the DEM. This is about the resolution of the WV-1 
imagery at 0.7 meter GSD. The following figure (Eigure 28) shows the DEM grid 
displayed over the WV-1 image (every 5* point has been displayed for clarity). 
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Figure 28. Portion of the WV-1 stereo derived DEM grid (every 5^*’ post displayed) 
showing the density of the grid (0.8 meter point spacing) 

The complete collection of the Socet Set NGATE automatic DEM point 
extraction is shown in the following figure (Eigure 29). 
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Figure 29. WV-1 stereo pair collected DEM (0.8 meter point spacing) auto 

collected from stereo pair 214/228 using Socet Set NGATE software 
(2,141,400 total points). 

The next step in the process was to take this stereo derived WV-1 DEM and 
generate a point cloud. This was accomplished by exporting the point cloud as an 
American Standard Code for Information Interchange (ASCII) DEM and importing this 
ASCII DEM into QT Modeler as a point cloud. The following figure (Eigure 30) shows a 
display of that point cloud. 
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WV-1 stereo derived DEM converted to a point cloud to be used 

for LiDAR fusion. 


Similar to the high resolution image DEM extraction from aerial photographs, the 
WV-1 point cloud has many errors near building edges and near trees. No manual editing 
was performed and the data were used as a pure autocorrelation data set. At this juncture, 
the DEM point cloud data was ready for EiDAR fusion. 

D. CHANGE DETECTION USING LIDAR AND STEREO POINT CLOUDS 

Eusion of EiDAR point cloud data and stereo point cloud data has many 
applications as outlined in the Introduction and Background chapters of this thesis. This 
research focused on two of those applications in the form of point cloud change detection 
and a fused Image-EiDAR product capable of measuring precision 3D coordinates. Eor 
change detection to occur, the EiDAR point cloud and stereo image point cloud data 
sources must be coincident. The previous sections in this thesis highlight methods to 
achieve this coincidental alignment by using the EiDAR intensity image and EiDAR 
DEM as a control source for the stereo imagery. The registration of the stereo imagery to 
the EiDAR data results in a stereo derived DEM that is coincident with the EiDAR point 
cloud. 
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Change detection can be an important tool to rapidly assess damage due to a 
natural or man-made event. Natural events like hurricanes, earthquakes, tsunamis, or 
fires can be assessed with before event and after event feature level data like a LiDAR 
point cloud or stereo derived point cloud. The following examples of the NFS campus 
data and the Pacific Beach data demonstrate how change detection can be used for 
analysis in spite of potential noise in the data sets. 

1. NFS Campus LiDAR and Stereo point cloud change detection 

Before beginning the change analysis, it was important to inspect and visualize 
the data. The following figure (Figure 31) show a side-by-side view of the LiDAR point 
cloud and the Stereo point cloud for comparison of quality. 
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Figure 31. Split screen close up view of the LiDAR data on the left and the 

stereo image collected point cloud on the right showing many unique differences. 
The LiDAR data is very clean and sharp (Left) and the Stereo point cloud is 
interpolated and rough in areas (Right). 


It proved to be very difficult to get a good, clean point cloud from the optical 
imagery even though the imagery was high quality. Notice how well the LiDAR point 
cloud data modeled the trees as can be seen by the realistic display of the tree in the 
center of the left image of Figure 31. Also, notice how clean the building comer lines are 
for the LiDAR (Figure 31, center red building on left image) and how noisy and 
sometimes interpolated the building lines are in the stereo cloud (Figure 31, center red 
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building on right image). Stereo image autocorrelation in tree canopy is very 
challenging. As was mentioned before, the stereo point cloud data was not manually 
edited but used as a pure autocorrelation data set. 

The next step was to come up with a method to compare these two data sets for 
change detection in spite of the differences already noticed from the autocorrelation 
errors. A straight subtraction method was used (data set 1 minus data set 2) to create a 
new point cloud model for display and analysis. Figure 32 shows the results of the new 
point cloud created by subtracting the LiDAR point cloud from the stereo point cloud. 
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Figure 32. Difference between the WSI LiDAR and Stereo Imagery point 
cloud Mean: -0.5m; Std Dev: 6.3m; Max: 44.0m; Min: -30.7m 
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Location: 36.596 N 121.874 W 


Results appear very noisy but predictable. The trees and building edges have the 
most noticeable differences as indicated by the red and blue coloration. Even though the 
differences look catastrophic, there are some good things in here. For instance, building 
tops, like the ground surface, have not changed. 

The results of the autocorrelation and the errors that could potentially exist on 

either data set required approaching the problem from a different perspective. Instead of 

focusing on the differences, focusing on the things that stayed the same seemed like a 

better approach. A high percentage of the autocorrelation data was very good with 

44 






autocorrelation performing well on the ground features and building tops. The new 
approach strategy was to keep the data that were similar and eliminate the noisy outlier 
data. While there are a number of ways to implement this, the above approach was 
accomplished for this research by using software called Cloud Compare (CloudCompare, 
2004) and clipping the difference cloud that was created in QT Modeler. The following 
figure (Figure 33) is a visualization of the original difference cloud in the Cloud Compare 
software. 



Figure 33. Vertical display of the point cloud created from the difference between 
the NFS WSI LiDAR and the Stereo Image autocorrelation point cloud 
using Cloud Compare (CC) software. 


Notice how the majority of the cloud differences shown in figure 33 are near zero 
(green). The analysis focused on how much of the scene stayed the same and used a 
difference histogram to capture the statistics. The scale bar shows the height difference 
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values (in meters) eompared to the eolors. A small, “on edge” vertieal histogram to the 
right of the seale bar highlights the fact that a majority of the differences are biased by 
about one meter. 

The following figure (Figure 34) represents only those objects that remain the 
same. The technique is to remove the change outliers, keeping only things that stayed in 
the same location. The change outliers are extreme differences caused by stereo model 
auto-correlation inaccuracies, sensor perspective differences, and normal time variant 
change (cars, building construction, tree removal, etc.). 
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Figure 34. Display of the first cut edit of the difference data clipped at plus and 
minus 2 meters to remove large outliers. The “holes” in the data are 
where outlier errors have been removed. 


The color image and color scale bar show how much things are the same between 


the LiDAR point cloud and the Stereo Image derived point cloud. The difference data 
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have been clipped at plus and minus 2 meters from the mean to remove the outliers. The 
“holes” are where the outliers existed. Amazingly, 34% of the points have been clipped 
but the contents of the scene are still discernible (Figure 34). 

By further clipping the outlier free data using the statistical 95% confidence 
interval equation (mean plus or minus two times the standard deviation), subtle 
differences are detected - almost at the sensor characteristic level. 
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Figure 35. Display of the difference data after further clipping at the 
95% confidence level. The differences are on the order 
of 50 cm (.5 meters) from the mean and show peculiar sensor collection 
level artifacts (red and blue areas). 
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In Figure 35, notice the 50 cm difference (red) in the driveway on the upper right 
of the image (Location 1 identified with the red arrow). Next, there is a -50 cm 
difference (blue) in the walkway on extreme upper right of the image (Location 2 
identified with the red arrow). Notice that the southeast facing roof sections of buildings 
show a 50 cm difference (Location 3 identified with the red arrow). These are all 
examples of extremely small differences that are sensor collection dependent or stereo 
extraction dependent. 

The important point is that the buildings and many of the trees remained the same 
even though quite a bit of noise due to errors in stereo correlation existed. Focusing on 
things that remained the same by clipping the gross error outliers has allowed analysis to 
take place at whatever level the errors exist at. The data showed that no extreme 
differences had occurred. The interesting thing about this analysis was that the final 
clipped difference data set (90% confidence interval) was able to find differences at the 
sensor collection level. 

2. San Diego LiDAR and Stereo WVl point cloud change detection 

Visualization and inspection of the data was performed in a similar fashion to that 
performed with the NFS data. The following figure (Figure 37) shows a side-by-side 
comparison of the LiDAR point cloud and WV-1 stereo extracted point cloud. 



Figure 36. Side-by-side zoomed in view of the Pacific Beach, CA, 

LiDAR point cloud (left) and WV-1 stereo point cloud (right). 
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The LiDAR point cloud is clean and sharp while the stereo point cloud shows 
areas of interpolation and missing features. As rough as the WV-1 stereo extracted point 
cloud was, it was used as a pure autocorrelation data set in the following analysis. 

The following figure (Figure 38) is the “first cut” removal of the outliers using a 
plus and minus 10 foot threshold around the approximate minus 4 foot bias. Fifteen 
percent (15%) of the point have been clipped as a result of this outlier removal, what 
remains are things that have not changed dramatically. 
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Figure 37. Cloud Compare software created figure of the entire 
Pacific Beach, CA, point cloud comparison test area 
showing the results of the “first cut” removal of 
difference outliers (85% unchanged, 15% change outlier). 
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Focusing on the difference data that did not change with the San Diego LiDAR 
and WV-1 stereo imagery presented an unexpected surprise. The outlier and 90% 
confidence interval clipping process detected an area of large scale change that 
corresponded to new home construction during the time of the LiDAR collection (2005) 
and the WV-1 stereo image collection (2008) (Figure 37). The area that has changed, in 
the upper right comer, stands out very noticeably as rectangular blue areas (Figure 37, red 
box). This is seen more clearly in Figure 38. It is easy to pick out the 7 houses in the 
upper right comer of Figure 37 and central part of Figure 38 that have been built between 
the 2005 to 2008 time frame (normal - time variant - change). Notice that even with the 
predominant noise around the homes and in the tree areas, the real change is easily 
noticeable as large “holes” in the data. It is also easy to pick out the taller building 
structures because of the auto-correlation outliers along the building edges that have been 
removed. 
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Figure 38. A closer inspection of the new home construction showing the 
results of the “first cut” outlier removal. The “holes” in the 
data show where new homes were built. 


The detection of this new home construction was unexpected and did show that 
the outlier removal process does work to detect large scale change. The key is to focus 
on the things that stayed the same, clip out the noise, and what you are left with is a data 
set that can show large change if it exists. 

E. PRECISION IMAGERY-LIDAR FUSED PRODUCT 

The next logical step is to focus on selected aspects of creating a fused optical 
image and LiDAR data product viable for extraction of precision 3D coordinates. This is 
different than assigning a Red-Green-Blue (RGB) value to each LiDAR point as is 
commonly done with LiDAR visualization tools and is supported by the standardized 
LiDAR file format (.las). This concept is innovative in that the idea is to preserve the 
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perspective view of the image, preserve the native resolution of the image, and use the 
coincident LiDAR data to facilitate precision 3D coordinate extraction (Rodarmel et ah, 
2006). 

The best way to approach this concept is to first build a stereo image base layer 
that has been registered to LiDAR. The reverse of this can be done, which would be 
registering the LiDAR to the base layer if the stereo image base layer is considered to be 
more accurate. In either case, a stereo image base layer and LiDAR data set should exist 
and should be registered such that they are coincident. Having these two data sets 
coincident provide for many, very attractive options. 

One option is that LiDAR data can be converted to a high resolution DEM and 
displayed, as individual points, in the stereo imagery. This provides measurable quality 
assurance for the LiDAR data as well as the stereo image models. 

A second option is that if the stereo imagery has a stated accuracy, the LiDAR 
data accuracy can be either verified or computed. For automated accuracy determination, 
the LiDAR accuracy can be computed by using an autocorrelated stereo image point 
cloud to LiDAR point cloud change detection process outlined in this thesis. Even with 
the noise of the stereo autocorrelation, a statistical change can be calculated allowing 
absolute and relative accuracy to be computed. 

In the most likely scenario, a new, more current and detailed image would be 
collected that would need to be registered to the EiDAR data. In this case, the stereo base 
layer provides for an attractive auto-registration layer that could be used to adjust the new 
image. Once coincident with the stereo base layer, the image can then be easily fused 
with the EiDAR data as it too is coincident with stereo base layer. 
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V. SUMMARY 


LiDAR point cloud data, imagery, and stereo image derived point cloud data are 
ideal candidates for data fusion to provide high precision digital elevation models and 
improve change detection. In order for accurate fusion to occur, however, the LiDAR 
and imagery must be coincident. 

This research demonstrates the importance of registering optical imagery to 
LiDAR data as a basis for proper, seamless fusion. Several dissimilar image data types 
were chosen to test optical image to LiDAR registration methods, including an airborne 
panoramic film image (OBC), high resolution stereo frame color images (UltraCam 
Eagle), and panchromatic satellite images (WVl). Likewise, several LiDAR sets of 
varying point densities and quality were chosen including LiDAR of Haiti at 3.4 
points/m^, the NFS Campus at 30.0 points/m^, and a portion of San Diego, California 
LiDAR at 1.4 points/m^. 

Individual points in a LiDAR point cloud are not easy to precisely identify on an 
image, consequently, methods for optical image to LiDAR registration were researched 
with the realization that the most practical method is to use the LiDAR-derived intensity 
image as an X,Y (Latitude, Longitude) control base and the LiDAR-derived DEM as the 
Z or height control base. A logical control point selection method was established that 
used features that were homogeneous in surrounding height. This helped with extracting 
a more precise height at a given X,Y location because the height could be interpolated 
from the EiDAR derived DEM. 

A new and robust process for registering optical imagery to EiDAR was 
developed as part of this research. This new method uses a two-step triangulation 
process that derives ground control from the EiDAR intensity image in one step and then 
adjusts the imagery in a second step. The advantages of this new method are that all 
measured points are fully reviewable and the initial triangulation process provides quality 
assurance of the intensity image control points. 
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These new methods and processes were applied to data for three test sites. In the 
first test scenario, OBC image adjustment to LiDAR for an area in Haiti proved to be 
problematic using the panoramic math model as the image and control points did not 
match well (RMS 1.9 meters). A better approach used the RPC representation of the 
panoramic math model followed by an affine transformation to fit the image to control 
points, with an improved RMS of 0.5 meters. The well-registered results permitted high 
resolution visualization of the OBC imagery and Haiti LiDAR point cloud. The RPC 
adjustment approach should help to streamline future OBC image and LiDAR fusion 
processing and bring more utility to the OBC imagery, as RPC adjustments are supported 
by more software packages than rigorous panoramic math models. The OBC imagery 
was not in stereo (a single image) so no further point cloud analysis was performed with 
this data set. 

Fusion of high resolution optical aerial imagery and dense LiDAR data of a 
portion of Monterey, California, over NPS campus was very straightforward. This well 
calibrated imagery had very good positional and attitude information, resulting in 
precision stereo pairs that needed no triangulation to remove parallax or control points to 
register the imagery to the LiDAR. The data sets were virtually coincident from the start, 
resulting in easy fusion of the imagery and LiDAR point cloud. The high resolution 
nature of the imagery and LiDAR data allowed for excellent 3D visualization of the NPS 
campus. 

For the NPS dataset, a high resolution DEM was automatically collected from the 
stereo images and this DEM was converted into a point cloud. Several automatic DEM 
extraction strategies were evaluated in the attempt to create a quality DEM. After these 
strategy tests, it was realized that autocorrelation errors will always be present and the 
perfect stereo collected point cloud cannot be obtained from autocorrelation alone. 

The NPS EiDAR point cloud and the stereo point cloud were then used as 

separate entities to create a change detection layer. A new point cloud was created that 

contained the difference between the EiDAR point cloud and the stereo point cloud. 

Using this “difference cloud”, a unique strategy was devised to focus on things that 

stayed the same as opposed to focusing on things that were different. This strategy used 
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a 90% confidence interval bracketing method that clipped outliers and autocorrelation 
errors. Surprisingly, this method worked well to show that buildings, objects, and trees 
were generally unchanged even though 30% of the points were noise caused by stereo 
image autocorrelation errors. 

The final test case investigated WV1 stereo image registration to LiDAR data and 
stereo image derived point cloud versus LiDAR point cloud change detection for the San 
Diego, California, site. The improved registration process was used to triangulate 
(remove y-parallax) and register (control) three WVl images to the LIDAR intensity 
image and LiDAR DEM resulting in a good registration solution (RMS 0.6 meters). 

Following registration of the three WVl images, several autocorrelation strategies 
were attempted with the stereo imagery to try to create the perfect point cloud. The best 
WVl autocorrelation point cloud, however, was still very noisy. A difference point cloud 
was created using the 90% confidence interval bracketing method to clip outliers and 
autocorrelation errors, resulting in detection of unforeseen urban change in the form of 
new home construction in the image. 

A short description of the concepts for creating a fused LiDAR/Image product 
was presented focusing on the requirement of using an accurate and coincident stereo 
image base layer as the foundation. Accuracy can be computed for the LiDAR data using 
the same stereo image point cloud to LiDAR point cloud change detection process 
outlined in this thesis regardless of the high potential for stereo autocorrelation errors. If 
a new, more current and higher resolution image is collected, the stereo base layer can be 
used in a photogrammetric auto-registration routine to align the new image with the 
stereo imagery with the added result of being coincident with the LiDAR data for easy 
fusion. 
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VI. CONCLUSIONS 


This research focused on demonstrating the processes and techniques of 
registering photogrammetric imagery to LiDAR point cloud data, and the importance of 
this registration as the precursor for data fusion. Point cloud data derived from optical 
stereo images were compared to LiDAR point cloud data and basic change analysis was 
performed. This work is a precursor to a larger initiative to create a “ready to use” 
photogrammetric image and LiDAR fused data set. 

Most image to LiDAR registration processes utilize edge matching techniques, 
however, this requires identifiable object edges throughout the image and sophisticated 
software to identify and extract those edges. The LiDAR intensity image approach 
demonstrated here proved more capable as it did not require the image to contain object 
edges which allowed registration of many image types with varying scene content. The 
intensity image approach was also easily adapted from known image-to-image 
registration approaches and a new procedure was developed for this process. A unique 
individual point selection technique that requires the point to be in a homogeneous height 
area was successful because it minimized the amount of derived LiDAR DEM height 
interpolation error. 

A new, 2-step triangulation process coupled with an RPC math model approach 
was used to improve the registration of the OBC panoramic imagery to LiDAR data. 
Significant improvement was achieved, from an RMS of L9m using the rigorous model 
to an RMS of 0.5 meters using the RPC method. The RPC adjustment approach should 
help to streamline future OBC image and LiDAR fusion processing and bring more 
utility to the OBC imagery, as RPC adjustments are supported by more software 
packages than rigorous panoramic math models. For all image types, the 2-step 
triangulation had the added benefits of retaining registration point measurements for 
quality review and intermediate control point derivation output for quality assurance. 

High resolution DEM extraction from stereo imagery can have as much as 30% 
autocorrelation failure principally near building edges and tree canopy. This high rate of 
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error was overcome when performing stereo image and LiDAR point cloud change 
analysis by clipping the outliers by a 90% confidence interval bracketing method to 
create a new point cloud by differencing the LiDAR point cloud and the stereo image 
point cloud. The display and analysis of the difference point cloud resulted in 
measurable change detection. This technique produced results that detected small scale 
(<50 cm) differences and unforeseen large scale change in the form of new home 
construction. 

Finally, a concept for a fused image and LiDAR data set was presented that 
focuses on creating a stereo image base layer that is coincident with the LiDAR data. 
This concept has many benefits including stereo image and LiDAR accuracy verification, 
LiDAR accuracy computation, and easier succeeding image/LiDAR registration and 
fusion. This research and these benefits justified the proposal for a fused LiDAR and 
stereo base layer as the foundation for further LiDAR/image fusion. 
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